Document Length Normalization

In the previous lecture we discussed pivoted document length normalization [Singhal et al. 96], a simple technique that applies a correction for the observation that document relevance correlates with document length. Through careful empirical verification of previous assumptions, they showed that the seemingly simple normalization term could have a big impact on results. However, in our discussion of the analysis that led to pivoted document length normalization, we passed over a basic question: How were the relevance judgments in the TREC dataset made on the approximately 740,000 documents and 50 queries?